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Objectives 
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■  Present  a  Case  Study 

■  A  CMMI®  Level  5  Assessment 

■  Focusing  on  a  Team  Software  Process  (TSP)SM  Team 

■  Provide  Guidance  for  TSP  Teams 

■  Does  TSP  Guarantee  CMMI  Level  5? 

■  How  can  TSP  be  used  to  support  a  CMMI  Level  5 
Assessment? 

■  What  did  this  team  do  to  meet  CMMI  Level  5 
objectives? 


The  audience  should  be  familiar  with  the  concepts  of  the  TSP  and  the  Persona!  Software  Process  (PSP)SM 


MMt  s  a  Roistered  Tradfi-m^k  af  Cine^ie  Mellon  University 
^Team  Software  Process,  TSP,  Personal  Software  Pvooess,  avJ  PSPaieService  Marks  of  Carnegie  Mellon  University 
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TSP  Supports  CMMI 
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In  the  late  1980s  and  early  1990s  the  SEI  developed  the 
Capability  Maturity  Model  (CMM)  which  captured 
organizational  best  practices  for  software  development 

SEI  Fellow,  Watts  Humphrey,  decided  to  apply  the  underlying 
principles  of  the  CMM  to  the  software  development  practices 
of  a  single  developer 

The  result  of  this  effort  was  the  Personal  Software  Process 
(PSP),  designed  to  be  a  CMM  level §.  process  for  individual 
software  developers 

Humphrey  then  developed  the  Team  Software  Process  (TSP) 
for  the  smallest  operational  unit  in  most  organizations,  the 
project  team.  TSP  was  designed  to  be  a  CMM  level  5 process 
for  project  teams. 


Source:  TSP  and  CMMI:  A  Brief  History  (http://wwrt.sei.cmu.edu/tsp/history.htnnl} 
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SEI’s  TSP  to  CMMI  Mapping  - 1 


OGDEN  AIR  LOGISTICS  CENTER 


T5P  and  CMMI  Process  Categories 
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Source:  James  McHale  and  Daniel  S,  Wall  Mapping  TSP  to  CMMI  ( CM  U/S  El  -  20  04- TR  -  0 14 ) 
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SEI’s  TSP  to  CMMI  Mapping  -  2 
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Process  Management 
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SEI’s  TSP  to  CMMI  Mapping  -  3 


Engineering 
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309  SMXG  Background 


Support  Staff 
7% 


Administration 
2% 

Technicians 
9% 


Configuration 

Management 

9% 


En  g  i  nee  ra'Comp  uter 
Scienti^s 
73% 


OGDEN  AIR  LOGISTICS  CENTER 

Large  Cadre  of  Talented 
People: 

■  700+  Civilian  Personnel 

■  Average  over  1 0  years 
technical  experience 

■  Growing  by  ~50  PEs/Year 
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309  SMXG 

Process  Improvement  History 

ogde  n  a  ir 


■  Focused  on  process 
improvement  since  1991 

■Assessed  in  1998  to  be 
Capability  Maturity  Model 
(CMM)  -  Level  5 

■Earned  AS9100  &  ISO  9001 
Registration  in  2006 

■Assessed  in  2006  to  be 
Capability  Maturity  Model 
Integration  (CMMI)  -  Level  5 
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SMXG  &  CMM  /  CMMI 


1 
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Assessed 


1991  1992  1993  1994  1995  1996  1997  1998  2000  2001  2002 


2006 
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Focus  Project:  GTACS 


1 


■ 
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CMMI  Level  5 


Focus  Project  for  CMMI  Assessment 


Q  Q  Q  Q 
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CMMI:  Capability 
Maturity  Model  Integration 
improves  the 
organization's  capability: 
management  focus 

TSP:  Team  Software 
Process  Improves 
team  performance; 
team  and  product 
focus 

PSP:  Personal 
Software  Process 
improves  individual 
skills  and  discipline; 
personal  focus 
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GTACS  Overview 


OGDEN  AIR  LOGISTICS  CENTER 


■Ground  Theater  Air  Control  System  (GTACS) 

■  Modular  Control  Equipment  (MCE)  &  Tactical  Air 
Operations  Modules  (TAOM)  create  a  ground-based 
computer  network  systems  that  coordinate  radar  and 
communications  data  signals.  GTACS  modules 
coordinate  ground,  airborne,  and  naval  elements  to 
plan,  execute,  and  evaluate  joint  operations. 
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GTACS  MCE  Software  Updates 
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309  SMXG  provides  Module  Control  Equipment 
(MCE)  software  updates  for: 

■  New  Features: 

♦  Updates  to  MIL-STD 
Communications  Protocols 

•  User-Initiated  Switch 
Action  &  Display  Changes 

*  Interfaces  with  New  Weapons 
and  Systems 

■  Defect  Corrections: 

•Software  Development 
Defects 

•  Government  Acceptance 
Test  Defects 

•Field-discovered  Problem 
Reports 
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Sf  GTACSTSP  Tracking  Tools 

OGDEN  AIR  LOGISTICS  CENTER 

■GTACS  Uses  The  Process  Dashboard  Tool  to: 

■  support  individuals  and  the  team  in  using  high- 
maturity  processes  for  software  development 

■  simplify  the  work  involved  in  following  a 
high-maturity  process 

■  help  individuals  to  follow  a  defined  process  and 
collect  metrics  data 

■  improve  the  accuracy  of  collected  metrics 
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GTACS  &  TSP:  Productivity 

OGDEN  AIR  LOGISTICS  CENTER 

TSP  Schedule  Improvement 

■  Zero  missed  deadlines 

■  Zero  ‘returned”  workload 

TSP  Quality  Improvement 

■  Zero  integration  delects 

■  Zero  integration  rework 


TSP  Productivity 
Improvement 

■  417%  increase  in 
SLOC/Hour! 

■  Completely  recouped  TSP 
investment  cost  alter  about 
1  month! 
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GTACS  &  TSP:  Quality 


OGDEN  AW  LOGISTICS  CENTER 


■  TSP  Promotes  High  Quality 
Processes 

■  Personal  Reviews 

■  Team  Peer  Reviews 

•  Integrated  Systems 
Engineering  Peer  Reviews 

■  GTACS  Team  Achieves 
99.4%  Detect  Removal 
Before  Release 


QurililyFivfil* 

Defect  Removal  Profile 

/  \  50 


30 

20 


0 


Process  Yields 


Plan 

Artnal 

X  -■•r.-j.-t  :l cm|:dl:  |U1 

'JJ.l  X 

X  :  j/f  UM  If  it  'J.XX 

il.1% 

X  Bi.iu:  alu:I  Iuttfl-rtkdi  .fit  'J'J  UX 

'J'J.  -  % 

X  J.f:  j/f  System  Test  'J'J  'JX 

'J'J.  -  X 

6/10/2007- SSTC  2007 


BE  AMERICA  'S  BEST 


15 


— ■  309  SMXG  CMM  /  CMMI 
Assessments 

OGDEN  AIR  LOGISTICS  CENTER 

■  1998:  Assessed  SW-CMM  Level  5 

■  2001 :  Evaluated  Moving  to  CMMI  vs  Another  SW- 
CMM  Assessment 

■2003:  Began  CMMI  Transition 

■  2005:  Performed  Two  SCAMPI  B’s  on  Potential 
Focus  Projects 

■2006:  Performed  SCAMPI  A  on  Focus  and  “Depth” 
Projects 
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SCAMPI  Family  of 
y  Appraisals 
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breadth  of  tailoring  - 

SCAMPI  C:  provides  a  wide  range  of 
options,  including  characterization  of  planned 
approaches  to  process  implementation 
according  to  a  scale  defined  by  the  user 

SCAMPI  B:  provides  options  in  model  scope 
and  organizational  scope,  but  characterization 
of  practices  is  performed  on  implemented 
practices 


SCAMPI  A:  Is  due  most  rigorous  method,  and 
is  the  only  method  that  can  result  in  ratings 
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SCAMPI  B  #1  Results  - 1 

GTACS  Project  Only* 
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SCAMPI  B  #1  Results  -  2 

GTACS  Project  Only 

OGDEN  AIR 


LOGISTICS  CENTER 


Risk  Level 

Total  Risks 

Process  Risks 

Artifact 

Risks 

Document 

Risks 

Non-Team 

Risks 

High 

19 

1 

17 

0 

1 

Medium 

67 

15 

18 

6 

28 

Low* 

0 

0 

0 

0 

0 

Total 

36 

16 

35 

6 

29 

*Low  risks  were  not  categorized  in  tine  first  SCAMPI  B 


■  Definitions 

■  Risk  Level 

•  the  risk  that  the  final  SCAMPI  A 
Assessment  will  have  a 
negative  finding 

■  Process  Risks 

‘the  GTACS  team  had  no 
process  in  place 

■  Artifact  Risks 

•  the  assessment  team  found 
insufficient  artifacts  to  assess 


Document  Risks 

•  GTACS  process  documentation 
needed  to  be  updated 

Non-Team  Risks 

*  the  responsibility  ol  a  team 
other  than  the  TSP  team,  such 
as  the  group’s  SEPG  or  the 
GTACS  CM  Team 
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SCAMPI  B  #1  Actions 

OGDEN  AIR  LOGISTICS  CENTER 

Addressed  Causal  Analysis  &  Resolution  (CAR)  Findings 
(High  Risk) 

■  Focused  on  executing  CAR  process 

■  Gathered  data  /  created  CAR  artifacts 

■  Process  Framework  &  Documentation 

■  Identified  missing  elements  (Requirements  Management,  Project 
Planning,  Project  Monitoring  &  Control) 

■  Created/Implemented  Process  Improvement  Proposals 

■  Team  Resources 

■  No  special  resources  devoted  to  CMMI 

■  Work  was  done  by  the  team  and  led  by  the  team’s  process 
manager  (a  standard  TSP  role) 
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SCAMPI  B  #2  Results  - 1 

GTACS  Project  Only* 
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SCAMPI  B  #2  Results  -  2 
GTACS  Project  Only 


Risk  Level 

Total  Risks 

Process  Risks 

i 

Artifact 

Risks 

7  OPEN  AIR  LOt 

Document 

Risks 

HSI1CS  CENTE 

Non-Team 

Risks 

High 

23 

0 

S 

0 

15 

Medium 

3s  ~1 

7 

6 

17 

S 

Low 

22 

0 

i 

ii 

10 

Total 

S3 

7 

15 

2S 

33 

■  Decision  Analysis  &  Resolution 
(DAR): 

■  7  High,  Artifact 

■  3  Medium,  Artifact 

■  1  Low,  Document 


■  Organizational  Process 
Performance  (OPP): 

■  13  High,  Non-Team 

■  t  Medium,  Non-Team 

■  f  Low,  Non-Team 


■  Training: 

■  1  High,  Artifact 

■  2  High,  Non-Team 

■  12  Medium,  Document 

■  f  Low,  Artifact 

■  f  Low,  Non-Team 

■  Others  with  GTACS 
responsibility: 

■  18  Medium  - 17  Low 

■  Spread  throughout  model 

■  QPM,  CAR,  and  Risk 
Management  required  process 
changes 
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DAR  Findings 


OGDEN  AIR  LOGISTICS  CENTER 


■  Scope 

■  During  1st  SCAMPI  B,  DAR  was  focused  at  the  organizational 
level 

■  This  proved  to  be  insufficient  to  produce  the  necessary  artifacts 

■  An  Organization  level  DAR  process  was  developed  by 
representatives  throughout  the  309th 

■  The  new  DAR  process  became  a  requirement  for  GTACS 

■  Timing 

■  The  majority  of  the  time  between  the  first  and  second  SCAMPI 
B’s  was  spent  at  the  Organizational  level 

■  GTACS  defined  its  process,  conducted  training,  but  did  not 
execute  the  process  prior  to  the  second  SCAMPI  B 
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DAR  “Supported”  by  TSP 

OGDEN  AIR 


LOGISTICS  CENTER 


IS?  and  cm  I  MLS 


PPSM!5 


■  SEI’s  Report:  All  DAR  Specific  Practices  ‘Partially  Addressed” 

■  Underestimates  Risk  and  Effort  for  TSP  Teams 

■  TSP  is  consistent  with  DAR  philosophy  but  is  nowhere  near  sufficient 

■  TSP  as  it  now  stands  is  insufficient  to  pass  a  CMMI  assessment 

■  TSP  DAR  procedure  is  required  to  produce  proper  and  meaningful 
artifacts 
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DAR  for  GTACS  TSP  Team 


OGDEN  AIR  LOGISTICS  CENTER 


Institutionalization 

■  The  309th  provided  DAR  training  for  GTACS 

■  GTACS  adapted  the  Organization  level  DAR  process  for  use  on 
their  team 

■  Determined  to  use  the  TSP  approach 

■  Scripts 

•  Forms 

■  Adaptation 

■  GTACS  created  a  draft  operational  process  in  the  form  of  a  TSP 
script 

■  The  DAR  script  was  then  used  to  analyze  several  different  types 
of  issues 

■  product  design 

•  tool  selection 

•  process 
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DAR  TSP  Script  for  GTACS 


DAR  Process  Script 
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Purpose 

To  guide  the  team  in  making  formal  decisions. 

Entry 

Criteria 

Either 

-  A  Critical  measurement  exceeds  the  thresholds  defined  in  the  GTACS  DAR  threshold  matrix 

or 

-  A  critical  decision  needing  a  formal  analysis  is  identified. 

General 

-  Critical  decisions  are  ones  that  have  potential  impact  on  the  project  or  project  team.  Issues  vith 
multiple  alternative  approaches  and  multiple  evaluation  criteria  are  particularly  well  suited  for 
fo  i  m  al  a  na  lysis.. 

Tailoring 

-  This  procedure  may  be  used  to  make  and  document  other  decisions 

Step 

Activities 

Description 

1 

Planning 

-  A  POC  is  assigned 

*  The  POC  may  be  sell  assigned  if  the  POC  is  responsible  for  the  critical  decision 

*  The  team  lead  assigns  the  POC  otherwise 

The  team  that  will  perform  the  DAR.  analysis  and  selection  activities  (the  DAR  team)  is 
assigned 

-  The  POC  completes  the  Entry  section  of  the  MXDE  Decision  Analysis  and  Resolution 

C oversheet  (section  I) 

*  The  completed  MXDE  Decision  Analysis  and  Resolution  Coversheet 

*  Scoringand  analysis  worksheets 

*  CM  is  notified  that  the  DAR  is  complete  and  that  the  DAR  artifacts  can  be 
archived  to  the  GTACS  data  management  repository 

Exit  Criteria 


-  The  MSDE  Decision  Analysisand  Rejoin  tion  Coversheet  is  completely  filled  out 

-  Tin?  artifacts  produced  during  the  DAR  activities  have  been  archived  in  accordance  with 


the  GTACS  DMF 


note:  Tlw  entire  process  script  can  be  found  in  the  Crosstalk  article  “CMMl  Level  5  and  the  Team  Software  Process" 
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OPP  and  QPM 
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■  GTACS  Weaknesses  in  OPP  and  QPM 

■  GTACS  data  were  not  analyzed  at  the  sub-process  level 

■  Data  analyses  did  not  address  an  understanding  ot  process 
variability 

TSP  Paradigm  is  Not  Consistent  with  CMMI  Expectations 

■  TSP  fundamentally  considers  the  software  development  process 
as  a  single  entity 

■  CMMI  expects  that  some  sub-processes  be  statistically  analyzed 
during  project  execution 

■  TSP  teams  use  data  for  three  purposes: 

1  project  planning  (whole  process) 

■  project  monitoring  and  oversight  (whole  process) 

■  process  improvement  (possibly  by  sub-process,  but  during 
Postmortems) 

■  TSP  uses  statistical  tools  such  as  Pareto  and  linear  regression 

■  CMMI  often  expects  run  charts  and  a  more  formal  statistical 
analysis 
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OPP  Solutions 


0  GDEN  AIR 


LOGISTICS  CENTER 


■  GTACS  Addressed  OPP  by 

■  Helping  the  Organization  address  Organizational  Process  Performance 
requirements  by  Organizational  Subprocess  (Process  Category) 

■  Mapping  detailed  TSP  project  data  to  Organizational  data  structures 

■  Using  Organizational  goals  to  guide  data  analyses 
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QPM  Solutions  - 1 
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■  Utilized  Existing  TSP  Data 

■  Earned  Value 

•For  project  management,  the  Projected  Completion  Date  is 
one  of  the  key  items  of  interest 

•GTACS  had  more  than  one  year’s  worth  of  statistically 
significant  data  on  Earned  Value 

■  Rework 

•TSP  teams  collect  a  great  deal  of  Defect  data,  including  time 
spent  in  the  finding  and  fixing  of  defects 

•GTACS  defined  Rework  as  the  percent  of  total  project  time 
spent  finding  and  fixing  defects  (consistent  with  the 
Organizational  definitions) 

•GTACS  had  more  than  one  year’s  worth  of  statistically 
significant  data  on  Rework 


6/19/2007- SSTC  200 7 


BE  AMERICA'S  BEST 


29 


QPM  Solutions  -  2 


Run  Chart  -  Forecast  Task  Completion 
with  control  limits 
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OGDEN  AIR  LOGISTICS  CENTER 


■  TSP  Postmortem  Process 

■  Currently  calls  tor  a  detailed  analysis  ot  project  and  process 
data,  including  identification  of  improvements 

■  This  provides  a  great  deal  of  support  for  CAR 

TSP  Postmortem  Lacks  CAR  “Formality”  and  “Feedback” 

■  CAR  artifacts  not  specifically  defined  nor  required 

■  No  examination  of  data  to  determine  if  implemented  process 
improvements  really  worked 

GTACS  Updated  the  Phase  Postmortem  Script 

■  Directly  addresses  Causal  Analysis  and  Resolution 

■  Created  a  CAR  Report 

•  Details  the  data  analysis  which  was  performed 

■  Identifies  any  weaknesses  discovered 

■  Suggests  process  improvements  to  address  these  weaknesses 

*  Requires  an  analysis  of  previous  project  improvements 
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CAR  Script 
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ISP  Postmortem  Script 

Purpose 

-  Toother,  analyze,  and  record  phice  and  project  data 

-  To  analyze  project  pejiaimanc  e  a^aiml  planiand  ^oali 

-  To  id e-  Jilidy  polenlnl  areai  for  j  mprevement 

-  To  ideality  and  mbmil  GCARi 

Entry  Criteria 

-  All  planned  work dor  the-  phxe  Joi  been  completed. 

-  All  project  and  pioceoi  dala  aie  available. 

-  Team  Dmhboajd  with  dala  ii  om  the  phxe  beinj;  analyied 

Piockl  Causal 
Analysis 


Quality  Causal 
Analysis 


The  tram  lead  leads  the  team  in  an  analysis  of  the  team  s  process  capability 

-  A  discumion  is  held  of  key  pieces  capabilities  (e.g..  estimation  accuracy, 
productivity,  quality)  and  compared  with  desired  numeric  goals 

-  A  determination  is  nude  to  identify  my  process  capability  deficiencies  that  exist  or 
if  there  are  process  capability  improvement  opportunities. 

Key  process  capabilities  are  analyzed 
Where  improvement  opportunities  are  identified 

Identify  process,  training,  tool,  support,  or  management  actions 
Submit  GCARs  as  required 


The  quality  manager  le  ads  the  team  in  an  analysis  of  the  team's  defect  data 

-  A  determination  of  which  defects  to  .analyze  is  made  through  an  analysis  of: 
Pareto  charts  for  the  nest  costly  .and  most  common  defect  types 

All  defect  escapes  from  Unit  Test 

-  For-  those  defects  and  defect  types  selected: 

Their  defect  log  entries  are  analyzed  to  determine  root  cause 
Where  improvement  is  needed 

Identify' process  (e.g.,  checklist^)  update),  training,  tool,  support,  or 
management  actions 
Submit  GCAKs  as  required 


iiion  L  Jie-Jd  ad  key  pioren  tapahihljeo  (e.gn  eitimalian  aaenney, 
productivity.  quality)  and  comp.ued  with  deified  numejjc  ijoali 
A  rUlftimi  nation  ir.  rmrU  In  irUnti+i.- a  riv  nraBuuajaakillin  rU+Lr  ii»ni-ii»-  I  Ini  mcir.t  m 
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CAR  Report 
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Training 
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■TSP  Training  Covers  Many  CMMI  Requirements 

■  Personal  Software  Process  Training  for  Engineers 

■  Managing  TSP  Teams  for  Managers  &  Team  Leader 

■  Addressed  the  training  requirement  (SP  2.5)  for: 

•  Project  Planning 

•  Project  Monitoring  &  Control 
•Integrated  Project  Management 

•  Process  and  Product  Quality  Assurance 

•  Measurement  &  Analysis 
•Causal  Analysis  &  Resolution 
•Verification  (partially  addressed) 

■A  Training  Plan  was  Developed  for  the  Remaining 
Process  Areas 
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Summary 
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TSP  Does  Not  Guarantee  CMMI  Level  5 

SEI’s  Mapping  of  TSP  to  CMMI  is  a  Good  Starting 
Point 

TSP  Teams  Should  Focus  on: 


■  Causal  Analysis  &  Resolution 

■  Decision  Analysis  &  Resolution 

■  Organizational  Process  Performance 

■  Quantitative  Project  Management 

■  Training 

■TSP  Does  Provide  All  the  Data  and  Discipline 
Required 

■Additional  Scripts,  Forms  and  Reports  are  Needed 
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Questions? 
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EL 
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9?  719  9  9 


9  9  9 
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Contact  Information 
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David  R.  Webb 
309  SMXG 

520  Software  Maintenance  Squadron 
Hill  Air  Force  Base,  Utah  84056 
(801)  940-7005 

E-mail:  david.webb@hill.af.mil 

Jim  VanBuren 

Draper  Laboratory 

517  Software  Maintenance  Squadron 

Hill  Air  Force  Base,  Utah  84056 

(801)  586-3346 

E-mail:  jim.vanburen@hill.af.mil 

Dr.  Eugene  Miluk 
Software  Engineering  Institute 
Carnegie  Mellon  University 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213 
(412)  268-5795 
E-mail:  gem@sei.cmu.edu 


CamegieMdlon 

Software  Engineering  Institute 
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Acronyms 
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309  SMXG: 

■  CAR: 

■  CM: 

■  CMMI: 

■  DAR: 

■  GTACS: 

■  MIL-STD: 

■  OPP: 

■  PSP: 

■  QPM: 
SCAMPI: 

■  SEI: 

■  SEPG: 

■  SP: 

■  TSP: 


309th  Software  Maintenance  Group 
Causal  Analysis  and  Resolution 
Configuration  Management 
Capability  Maturity  Model  Integration 
Decision  Analysis  and  Resolution 
Ground  Theater  Air  Control  System 
Military  Standard 

Organizational  Process  Performance 
Personal  Software  Process 
Quantitative  Project  Management 

Standard  CMMI  Appraisal  Method  for  Process 
Improvement 

Software  Engineering  Institute 
Software  Engineering  Process  Group 
Specific  Practice  (in  the  CMMI) 

Team  Software  Process 
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